Catch trial performance

## [1] "Excluded 11 participants based on catch-trial performance."

Exclusion of random guesses

We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.

## Warning in cor(rating_m, rating_m_overall): the standard deviation is zero
## [1] "Excluded 1 participants based on random responses."

Aggregated results

## [1] "Exposure speaker x block x first speaker type"

## [1] "Block x exposure speaker"

## [1] "Block x first speaker type"

## [1] "Block x first speaker type"

Individual responses

AUC computation

We use the AUC function with the splines method to directly compute the AUC.

t-test and regression model with control variables:

## 
##  Two Sample t-test
## 
## data:  aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = -0.31651, df = 180, p-value = 0.752
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -7.263266  5.255243
## sample estimates:
## mean of x mean of y 
##  10.83339  11.83740
## 
## Cohen's d
## 
## d estimate: -0.04692326 (negligible)
## 95 percent confidence interval:
##      lower      upper 
## -0.3394947  0.2456482
## 
## Call:
## lm(formula = auc_diff ~ cond + test_order + first_speaker_type + 
##     confident_speaker + block, data = rbind(aucs.cautious, aucs.confident))
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -43.974 -10.213  -0.943  10.862  46.588 
## 
## Coefficients:
##                                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                        47.070      4.818   9.770   <2e-16 ***
## condconfident (probably-biased)     1.271      2.597   0.489   0.6251    
## test_orderreverse                   3.092      2.605   1.187   0.2368    
## first_speaker_typeconfidentfirst    1.894      2.602   0.728   0.4677    
## confident_speakerconfidentm        -4.685      2.606  -1.798   0.0739 .  
## block                             -24.306      2.597  -9.359   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 17.52 on 176 degrees of freedom
## Multiple R-squared:  0.3451, Adjusted R-squared:  0.3265 
## F-statistic: 18.55 on 5 and 176 DF,  p-value: 8.889e-15
## Analysis of Variance Table
## 
## Model 1: auc_diff ~ cond
## Model 2: auc_diff ~ cond + test_order + first_speaker_type + confident_speaker + 
##     block
##   Res.Df   RSS Df Sum of Sq      F    Pr(>F)    
## 1    180 82409                                  
## 2    176 54003  4     28406 23.144 2.204e-15 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1